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Abstract 

This  paper  continues  the  study  of  a  file  search  problem  in  Monahan  [4].  The 
objective  is  to  determine  whether  or  not  a  record  is  in  a  sequential  file  whose  contents 
are  initially  unknown.  Costly  information  regarding  the  contents  of  positions  within  the 
file  can  be  purchased.  The  problem  of  determining  an  optimal  search  and  disposition 
strategy  is  formulated  as  a  Markov  decision  process  whose  state  space  is  polynomial 
in  the  number  of  possible  records  in  the  file.  As  a  result,  the  computational  effort 
required  to  determine  transition  probabilities  and  expected  payoffs  is  also  polynomial. 
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1  Introduction 

This  paper  continues  the  study  of  a  file  searching  problem  analyzed  in  Monahan  [4].  The 
file  consists  of  a  subset  of  records  that  have  been  randomly  selected  from  a  given  population 
of  records.  The  probability  of  selecting  each  record  in  the  population  is  known.  Records 
are  stored  sequentially  on  the  basis  of  some  key  value.  We  are  concerned  with  whether  a 
given  record  is  or  is  not  in  the  file.  We  can  declare  the  status  of  the  record  immediately 
or  we  can  gather  costly  information  about  the  contents  of  the  file  by  examining  as  many 
positions  within  the  file  as  we  wish  and  then  making  the  declaration.  A  reward  is  earned  if 
this  declaration  is  correct.  The  objective  is  to  determine  a  strategy  for  acquiring  information 
and  making  a  declaration  that  maximizes  the  expected  terminal  payoff  less  the  total  cost  of 
acquiring  information. 

Monahan  [4]  models  this  problem  as  a  Markov  decision  process  (MDP)  whose  state 
space  is  the  set  of  probability  distributions  over  the  set  of  all  possible  file  contents.  In  this 
paper,  we  propose  another  MDP  whose  state  space  is  considerably  smaller,  which  leads  to  a 
polynomial-time  algorithm  for  solving  the  problem. 

The  paper  is  organized  as  follows.  An  explicit  statement  of  the  problem  is  given  in  the 
next  section.  Section  3  discusses  the  relationship  between  the  model  developed  here  and 
other  models  of  search.  The  reduced-state  MDP  is  given  in  Section  4.  Section  5  derives 
explicit  expressions  for  the  state  transition  probabilities.  A  numerical  example  is  given  in 
Section  6.  Concluding  remarks  are  in  Section  7. 

2  The  Problem 

There  are  n  possible  records,  each  uniquely  labelled  by  an  integer  between  1  and  n.  The  file 
includes  record  i  with  probability  pt.  The  selection  of  any  record  is  stochastically  independent 
of  the  selection  of  other  records.  The  members  of  the  file  are  sorted  with  the  first  position 
in  the  file  being  the  smallest  included  record.  For  example,  the  probability  that  the  first 
position  contains  record  3  is  (1  —  pi)(l  —  P2)Pz-  The  total  number  of  included  records 
is  a  random  variable  (which  is  the  sum  of  n  Bernoulli  random  variables  with  parameters 
Pi, . . .  ,pn,  respectively),  whose  value  is  initially  unknown. 


A  query  asks  for  the  contents  of  the  7-th  position  in  the  file,  which  is  the  7-th  smallest 
included  record.  It  costs  $c  to  make  each  query.  (Note:  Capital  letters  will  be  used  for 
positions  in  the  file.  Small  letters  refer  to  the  value  of  keys  of  records  in  the  file.  For 
example,  we  refer  to  record  h  in  position  I.)  We  are  either  told  which  record  is  in  position 
7,  or  that  fewer  than  7  records  have  been  included  (i.e.,  position  7  is  empty).  A  negative 
query  is  a  query  that  indicates  that  position  7  is  empty. 

We  wish  to  determine  whether  record  h  has  been  included  in  the  file.  At  any  stage  of 
the  decision  process,  we  make  a  query  or  terminate  the  search  by  either: 

I.  Claiming  record  h  is  in  the  file  but  not  specifying  a  specific  position  or 

II.  Claiming  record  h  has  not  been  included  in  the  file  and  that  there  are  exactly  L  records 
in  the  file  smaller  than  h. 

If  the  disposition  claim  is  correct,  a  reward  of  $r  is  earned. 

3     Related  Literature 

Monahan  [4]  discusses  the  relationship  between  the  file  search  model  studied  here  and  other 
search  models,  including  file  search  models,  that  are  in  the  computer  science,  economics, 
and  operation  research  literatures.  For  completeness,  the  highlights  of  that  discussion  are 
included  here. 

Wiederhold  [8]  describes  several  techniques  for  searching  a  sequential  computer  file  but 
makes  no  assumptions  regarding  the  likelihood  of  certain  records  being  in  the  file.  The  model 
studied  here  can  be  viewed  as  Bayesian  version  of  the  binary  search  and  probing  schemes  he 
discusses. 

There  is  an  extensive  literature  dealing  with  methods  for  searching  sequential  computer 
files.  See  Knuth  [2,  3]  for  a  discussion  of  "classical"  file  search  procedures.  The  classical 
problem  differs  from  the  problem  studied  here.  In  the  classical  problem,  the  records  in  a 
sequential  file  are  known.  A  record  is  randomly  drawn  with  a  known  probability  from  a  given 
population.  Its  key  cannot  be  observed.  The  objective  is  to  determine  if  this  record  is  in  the 
file.  The  only  way  to  make  this  determination  is  to  compare  the  record  to  existing  records 
in  the  file.   If,  after  a  comparison,  the  keys  match,  the  record  is  in  the  file.   If  they  do  not, 


additional  comparisons  are  made.  An  optimal  search  procedure  prescribes  the  sequence  of 
comparisons  that  optimizes  some  objective.  See  Moore,  et  al.  [5]  and  Moore  and  Whinston 
[6,  7]  for  a  decision-theoretic  discussion  of  this  form  of  the  problem. 

The  assumptions  regarding  what  is  known  and  not  known  are  reversed  in  the  problem 
studied  here.  We  have  one  record  whose  key  is  known  and  wish  to  determine  if  it  is  in  a  file 
consisting  of  records  whose  keys  are  initially  unknown. 

The  search  problem  studied  here  applies  to  settings  other  than  sequential  computer  files. 
Indeed  it  applies  to  any  problem  that  is  characterized  by  entities  randomly  drawn  from 
a  known  population  so  that  certain  of  them  may  or  may  not  be  present.  Those  that  are 
present,  however,  always  appear  in  a  known  order.  Possible  examples  include  the  videotape 
of  a  sporting  event  or  the  geologic  structure  of  a  potential  drilling  site  for  an  oil  well. 

4      A  Stochastic  Dynamic  Program 

We  wish  to  determine  whether  record  h  is  in  the  file.  (Monahan  [4]  refers  to  this  as  Problem 
h.)  We  have  examined  some  of  the  positions  in  the  file  and  know  that  records  i  and  k  are 
in  positions  /  and  I  +  M  +  1,  respectively,  with  i  <  h  <  k,  but  we  have  no  information 
regarding  the  M  records  between  i  and  k.  That  is,  record  i  (k)  is  the  greatest  (least)  record 
known  to  be  in  the  file  that  is  smaller  (greater)  than  h.  The  dynamic  program  developed  in 
this  section  exploits  the  fact  that  all  subsequent  decisions  need  only  take  into  account  i,A', 
and  M. 

The  decision  process  consists  of  two  phases.  In  the  search  phase,  positions  in  the  file  are 
examined  at  a  cost  of  $c  per  position.  At  any  stage  of  the  decision  process,  a  search  strategy 
specifies  the  next  position  to  be  examined  based  upon  all  of  the  information  available  at 
the  beginning  of  that  stage.  After  examining  a  position,  the  state  of  the  decision  process  is 
updated  to  reflect  the  information  that  has  just  been  obtained.  The  process  then  proceeds 
optimally. 

The  second  phase  of  the  decision  process  is  the  disposition  phase  that  entails  the  choice 
of  one  of  several  stopping  actions.  Based  upon  the  state  of  the  process,  record  h  is  declared 
either  to  be  in  the  file  or  not  in  the  file.  The  disposition  strategy  specifies  which  of  the 
stopping  actions  given  in  I  and  II  above  are  to  be  taken.  A  correct  disposition  decision  earns 


$r.  The  objective  is  to  find  a  search  and  disposition  strategy  that  maximizes  the  expected 
terminal  reward  less  the  total  cost  of  examining  the  file.  Such  a  strategy  is  called  optimal. 

We  formulate  the  problem  of  searching  a  sequential  file  for  record  h  as  a  stochastic 
dynamic  program  (Markov  decision  process).  For  notational  convenience,  the  dependence  of 
the  elements  of  the  model  on  h  is  suppressed. 

The  state  space  of  the  decision  process  is 

5=  {(i,k,M)\i  =  0,  ...,h-  1,  k  =  h  +  l,...,n  +  1,  and  M  =  0,  ...,k-i-  1}. 

We  add  to  the  original  n  possible  records,  the  fictitious  record  0,  which  is  smaller  than  all 
other  records.  We  specify  p0  =  1,  so  that  record  0  is  known  to  occupy  position  0  before 
any  queries  are  made.  Thus  a  state  with  i  =  0  indicates  we  have  not  yet  observed  any 
(non-fictitious)  record  smaller  than  h  in  the  file. 

By  convention,  k  =  n  +  1  labels  the  condition  where  we  have  not  yet  observed  any  records 
larger  than  h.  When  k  =  n  +  1,  M  denotes  the  maximum  number  of  records  that  could 
possibly  be  in  the  file.  Thus  the  initial  state  of  the  decision  process  is  (0,n  +  l,n).  If  our  first 
query  reveals  that  there  is  no  record  in  position  J,  we  will  then  be  in  state  (0,  n  +  1,  J  —  1). 

At  times,  it  will  be  convenient  to  say  that  record  n  +  1  (considered  larger  than  record 
h)  was  observed  in  a  given  file  position,  rather  than  saying  the  query  on  that  position  was 
negative. 

At  any  stage  of  the  decision  process,  one  of  several  actions  can  be  taken.  Suppose,  for 
example,  that  record  i  is  in  position  /  and  that  the  state  is  (z,  k,M)  G  S.  One  possible 
action  is  to  query  position  /  +  L,  for  L  =  1, . . . ,  M,  at  a  cost  of  $c.  Suppose  that  record 
j  is  observed  in  this  position.  The  new  state  of  the  decision  process  depends  both  upon  j 
and  the  current  state.  Potential  new  states  include  (j,k,M  —  L),  which  occurs  if  j  <  h, 
and  (i,j,L  —  1)  if  j  >  h.  (Note  that  this  is  consistent  with  our  conventions  in  the  case 
j  =  k  =  n  +  1.)  The  probability  that  the  query  in  position  I  +  L  shows  record  k  is  denoted 
by  p(j\i,  A:,M,  L).  This  is  a  transition  probability  between  the  state  (i,k,M)  and  either 
(j,k,M  —  L)  or  (i,j,L  —  1)  as  described  above.  In  the  next  section,  these  probabilities  are 
expressed  in  terms  of  pi, . . .  ,pn — the  underlying  probabilities  that  records  1, . . . ,  n  are  in  the 
file.  After  making  a  query,  the  decision  process  continues  for  at  least  one  more  stage. 

Other  actions,  all  of  them  associated  with  stopping  the  decision  process,  are  possible. 


When  a  disposition  action  is  taken,  the  expected  terminal  reward,  whose  value  depends 
upon  the  current  state,  is  received  and  the  decision  process  terminates.  Option  II  requires 
that  we  specify  the  "gap"  in  which  record  h  falls.  Suppose  that  the  state  is  (i,k,M)  G  <S 
and,  without  loss  of  generality,  that  record  i  is  in  position  /.  We  say  record  h  is  in  Gap  L 
if  the  record  in  position  I  -\-  L  is  less  than  record  h  and  the  record  in  position  I  +  L  -\-  \ 
is  greater  than  h,  for  L  =  0, . . . ,  M.  In  the  specific  case  in  which  all  queries  have  received 
negative  responses  {i  =  0  and  k  =  n  + 1),  record  h  is  said  to  be  in  Gap  0  if  the  file  is  empty 
or  if  all  records  in  the  file  are  greater  than  h.  This  is  consistent  with  Monahan  [4]. 

Let  V(i,  k,  M)  denote  the  maximum  expected  payoff  that  can  be  received  if  the  state 
of  the  decision  process  is  (i,  fc,  M)  £  S  and  the  process  continues  from  the  next  stage  on- 
wards optimally.  Then  V(-,-,-)  satisfies  the  following  dynamic  programming  recursion.  For 


V{i,k,M)  =  max  < 


where 


J(z,  k,  A/,  L)  expected  reward  if  the  L-th.  record 
following  record  i  is  examined  and 
the  process  proceeds  optimally,  L  = 
0,...,M 

S(i,k,M)  expected  (terminal)  reward  if  the 
search  process  is  terminated  and 
record  h  is  declared  to  be  in  the  file 

G(i,  k,  M,  L)  expected  (terminal)  reward  if  the 
search  process  is  terminated  and 
record  h  is  declared  to  be  in  gap  L, 
L  =  0,...,M 


h-\ 


J(i,k,M,L)    =    -c+~rp(h\i,k,M,L)+  J2  p(j\i,k,M,L)V(j,k,M  -  L) 


j-i+l 


+  Y,  PU\*,bM,L)V(i,j,L-l), 

J  =  h+l 


(l) 


(2) 


for  L  —  1, . . . ,  M.  The  first  term  in  J  is  the  cost  of  the  query.  The  second,  third,  and  fourth 
terms  are  the  optimal  expected  benefits  that  can  be  obtained  if  a  query  reveals  that  the 
record  in  "location"  L  is  equal  to,  smaller,  or  larger,  than  h,  respectively. 


Under  Option  I, 

M 

Under  Option  II, 

h-\ 


G(i,k,M,L)  =  rY^ 


j=i 


P(j\i,k,M,L)-  J2  p(t\J,k,M-L,l) 

t=h+l 

for  L  =  0, . . . ,  M. 

In  the  next  section,  we  derive  an  explicit  representation  for  the  transition  probabilities 
p(j\i,  k,  Af,  L)  in  terms  of  the  underlying  probabilities  pi,...,pn.  We  then  illustrate  this 
procedure  with  a  small  numerical  example  used  in  [4]. 

5      The  State  Transition  Probabilities 

Recall  that  p,  is  the  probability  that  record  i  is  included  in  the  file.  Let  qt ■  =  1  —  p,.  The 
probabilities  presented  in  this  section  are  computed  using  Bayes'  rule.  It  is  convenient  to 
consider  two  mutually  exclusive  cases  representing  information  about  the  file.  In  all  cases, 
we  assume  (z,  k,  M)  €  S  is  the  state  of  decision  process. 

5.1      Case  1:  k  <  n  +  1 

In  this  case,  we  have  observed  at  least  one  record  greater  than  h.  We  also  know  that  there 
are  exactly  M  records  between  records  i  and  k. 

Let  s(i,  k,  M)  denote  the  probability  that  records  i  and  k  are  in  the  file,  as  well  as  exactly 
M  of  records  i '  +  1  through  k  —  1.  These  probabilities  are  computed  recursively  as  follows: 

ifc-l  A:-l  j-\ 

s(i,k,Q)  =  pipk  q3      and     s(i, k1M-\-l)=pi   }(  qm)s(j,k,M), 

j=t+l  j=i+l    m=j+l 

with  the  convention  that  the  product  over  an  empty  index  set  is  1.  The  (posterior)  probability 
that  record  j  is  the  L-th  largest  record  in  the  file  following  record  i  is 

s(i,j,  L  -  l)5(j,  k,M  -  L)p~l 

Poia-.M,x)=  ^M) — &-.  (3) 


5.2      Case  2:  k  =  n  +  1 

In  this  case  record  i  <  h  is  the  largest  record  located  by  query  and  M  is  the  maximum 
number  of  records  that  are  greater  than  i.  (Since  there  are  n  possible  records,  M  <  n  —  i, 
with  inequality  holding  if  a  query  has  received  a  negative  answer.) 

For  g  <  n  + 1,  let  u(g,  TV)  denote  the  (prior)  probability  that  the  file  includes  record  g  and 
at  most  N  records  greater  than  g.  As  before,  these  probabilities  can  be  defined  recursively 
as  follows: 

n 

u{g,0)    =   Pg  Yl  V    and 

j=g+i 

nj  —  l  n 

u(g,N  +  l)    =    Pg   ]T  (    [J    ?m)«0',JV)  +  p„    J]   9i. 

j=g+l    m=g+l  j=g+l 

For  y  <  n+1,  the  (posterior)  probability  that  record  j  is  the  L-th  smallest  record  included 

in  the  file  after  record  i  is 

s(i,j,L  -  l)u(j,M  -  L)p~l 
u{i,  M) 

The  chance  that  the  query  in  position  L  is  negative  is: 

n 

p(n  +  1  |  i,  n  +  1,  M,  £)  =  1  -  ^  p{j  |  i,  n  -I-  1,  M,  L). 

6      A  Numerical  Example 

The  following  numerical  example  is  given  in  Monahan  [4]:  n  =  3,  pi  =  0.1,  P2  =  0.2,  p$  =  0.3, 
r  =  1,  and  c  =  0.1.  The  problem  is  to  determine  the  status  of  record  in  the  population  whose 
key  is  2,  so  that  h  =  2. 

There  are  a  number  of  standard  procedures  for  computing  the  solution  to  the  reduced- 
state  MDP  given  in  (1);  see,  for  example,  Heyman  and  Sobel  [1,  Section  4-5].  For  expositional 
purposes,  we  use  one  of  the  linear  programming  approaches.  Constraints  come  in  "bundles", 
one  bundle  per  state,  and  one  constraint  for  every  action  that  is  feasible  in  that  state.  At 
least  one  slack  variable  will  be  zero  for  constraints  in  a  given  bundle.  The  action  associated 
with  this  constraint  is  an  optimal  action  to  take  in  the  state  associated  with  the  bundle  in 
which  that  constraint  appears.  The  complete  formulation  is  given  in  Appendix  1. 


6.1      The  Optimal  Search  and  Disposition  Strategy 

We  know  that  following  variables  are  equal  to  one  since  they  correspond  to  states  representing 
complete  certainty  regarding  the  contents  of  the  file: 

V(0,4,0)  =  V(0,3,2)  =  V(l,3,0)  =  V(l,4,0)  =  K(l,3, 1)  =  V(0,3,0)  =  1. 

We  substitute  these  values  into  the  linear  program,  as  well  as  the  values  for  the  posterior  prob- 
abilities that  are  determined  using  (3)-(4)  and  obtain  the  explicit  form  of  the  linear  program 
given  in  Appendix  2.  In  this  formulation,  V(i,  k,  M)  is  written  as  Vikm,  for  (i,  fc,  M)  G  S. 

The  optimal  search  and  disposition  strategy  is  determined  by  the  value  of  the  slack 
variables  in  the  optimal  solution  to  the  linear  program.  For  convenience,  the  rows  whose 
slack  variables  are  zero  are  labelled  with  their  corresponding  state-action  pair. 

The  optimal  expected  payoffs  associated  with  states  in  S  for  which  there  is  uncertainty 
regarding  the  status  of  record  2  are: 

V031  =  V041  =  VUl  =  VU2  =  0.9,  V042  =  0.8299,  and  V043  =  0.89. 

The  following  table  lists  each  state-action  pair  for  which  the  associated  slack  variable  is  zero 
in  the  optimal  solution. 


Optimal 

Row 

State 

Action 

2) 

(0,3,1) 

L=  1 

5) 

(0,4,1) 

L  =  1 

9) 

(0,4,2) 

L  =  1 

14) 

(1,4,1) 

L  =  1 

17) 

(0,4,3) 

L  =  1 

23) 

(1,4,2) 

L  =  1 

Table  1:  An  Optimal  Search  and  Disposition  Strategy 

The  expected  payoff,  given  we  know  nothing  about  the  contents  of  the  file  and  follow  an 
optimal  search  and  disposition  strategy,  is  0.89,  since  the  initial  state  of  the  decision  process 
is  (0,4,3).  Using  Table  1,  the  optimal  search  and  disposition  strategy  is  the  following: 
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•  Examine  the  first  position  of  the  file.  The  initial  state  is  (0,4,3)  and,  from  Row  17  of 
Table  1,  the  optimal  action  to  take  is  L  —  1  (which  refers  to  the  position  1  in  the  file). 

•  If  a  "1"  is  observed  in  position  1,  examine  position  2.  Record  1  in  position  1  corresponds 
to  state  (1,4,2).  From  Row  23  in  Table  1,  the  optimal  action  is  L  —  1,  indicating  that 
it  is  optimal  to  examine  the  position  immediately  following  record  1,  which  is  position 
2.  An  examination  of  the  second  position  will  determine  the  status  of  record  2  with 
certainty. 

•  If  anything  other  than  "1"  is  observed  in  the  first  position,  the  status  of  record  2  is 
known  with  certainty. 

This  is,  of  course,  the  same  strategy  that  was  determined  by  a  different  procedure  in  [4]. 

7      Concluding  Remarks 

The  problem  of  searching  for  a  record  in  a  sequential  file  was  formulated  as  a  finite-state 
and  finite-action  Markov  decision  process.  We  refer  to  this  formulation  as  a  "reduced-state" 
model  since  the  number  of  states  in  the  state  space  is  polynomial  (0(n3))  in  the  size  of 
the  underlying  population.  It  is  easy  to  see  that  the  transition  probability  calculations  in 
Section  5  and  the  expected  payoff  calculations  in  Section  4  can  be  carried  out  in  polynomial 
time,  so  we  have  a  polynomial-time  algorithm  for  finding  the  optimal  strategy. 

More  general  versions  of  the  model  are  possible.  We  could  give  the  player  the  option 
of  identifying  the  exact  location  of  record  h  for  a  larger  reward,  or  claiming  the  record 
is  not  in  the  file  without  identifying  the  gap  for  a  smaller  reward.  The  state  space  and 
transition  probabilities  would  be  the  same  as  for  the  original  problem,  but  the  expected 
payoff  calculations  would  have  to  include  the  additional  options.  It  would  also  be  possible 
to  make  the  cost  of  queries  and  rewards  time-dependent  by  adding  a  time  dimension  to  the 
state  space.  An  interesting,  but  more  difficult,  generalization  would  permit  multiple  copies 
of  records  to  be  in  the  file. 


Appendix  1:  A  Linear  Program 

The  state  and  action  associated  with  each  constraint  is  noted  in  the  following  linear  program. 
For  notational  convenience,  s  is  used  to  represent  the  current  state  in  the  expressions  for  the 
transition  probabilities  in  each  block  of  constraints. 

Min       £      V(i,k,M) 

Subject  to: 
State  Action         Inequality 

s  =  (1,4,1)    1  =  1      7(1,4,1)  >     -c  +  p(2|s,l)  +  K3|s,  1)7(1,  3,0) 

+  (l-p(2|s,l)-p(3|s,l))7(l,4,0) 
In  7(1,4,1)  >     p(2|s,l) 

Gap  1      7(1,4,1)  >     1  -  p(2|s,  1) 

s  =  (0,4,1)     L  =  l      7(0,4,1)  >     -c  +  p(2|s,l)  +  p(l|s,  1)7(1,4,0) 

+  (l-p(l|s,l)-+p(2|s,l)-p(3|s,l))7(0,4,0) 

+p(3|s,  1)7(0,3,0) 

In  7(0,4,1)  >     p(2|s,l) 

GapO      7(0,4,1)  >     l-p(l|s,l)-p(2|s,l) 

Gap  1      7(0,4,1)  >     p(l|s,l) 

s  =  (0,4,2)     1  =  1       7(0,4,2)  >     -c  +  p(2|s,l)  +  p(l|s,' 1)7(1, 4,1) 

+  (1  -  p(l|s,  1)  -  p(2|s,  1)  -  p(3|s,  1))7(0, 4, 0) 
+p(3|s,  1)7(0, 3,0) 

1  =  2      7(0,4,2)  >     -c  +  p(2|s,2)  +  p(3|s,2)  7(0,3,1) 

+  (l-p(2|s,2)-p(3|s,2))  7(0,4,1) 

In  7(0,4,2)  >     p(2|s,l)  +  p(2|s,2) 

GapO      7(0,4,2)  >     l-p(l|s,l)-p(2|s,l) 

Gap  1      7(0,4,2)  >     p(l|s,l)(l-p(2|s,2)) 

s  =  (0,4,3)     L  =  l      7(0,4,3)>     -c  +  p(2|s,l)  +  p(l|s,l)7(l,4,2) 

+  (1  -  p(l|s,  1)  -  p(2|s,  1)  -  p(3|s,  1))  1/(0,4, 0) 
+p(3|s,l)7(0,3,2) 

L  =  2      1/(0, 4, 3)  >     -c  +  p(2|s,  2)  +  p(3|s,  2)  1/(0, 3, 1) 

+  (l-p(2|s,2)-p(3|s,2))7(0,4,l) 
1  =  3      1/(0,4,3)  >     -c  +  p(3|s,3)  1/(0,3,2)  +  (1  -  p(3|s,3))7(0,4,2) 
In  7(0,4,3)  >     p(2|s,l)  +  p(2|s,2) 

GapO      7(0,4,3)  >     1  -  p(l|s,l)  -  p(2|s,l) 

Gapl      7(0,4,3)  >     p(l|s,  1)  (1  -  p(2|s,2)) 
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State  Action         Inequality 


s  =  (0,3,1)     L  =  l  7(0,3,1)  >  -c  +  p(2|s,l)  +  p(l|s,  1)7(1,3,0) 

In  7(0,3,1)  >  p(2|s,l)  +  p(2|s,2) 

Gap  1  7(0,3,1)  >  p(l|s,l) 

s  =  (1,4,2)     L  =  \  7(1,4,2)  >  -c  +  p(2|s,l)  +  p(3|s,  1)7(1,3,0) 

+  (l-p(2|s,l)-p(3|s,l))7(l,4,0) 

L  =  2  7(1,4,2)  >  -c  +  p(3|s,2)7(l,3,l)  +  (l-p(3|s,2))7(l,4,l) 

In  7(1,4,2)  >  p(2|s,l) 

Gap  1  7(1,4,2)  >  l-p(2|s,l) 
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Appendix  2.  The  Explicit  Linear  Program. 

min  V031  +  V041  +  V042  +  VUl  +  V043  +  V142 

Subject  to: 

2)  1/031  >=  .9 

3)  KOSl  >=  .69231 

4)  V0S1  >=  .3077 

5)  V041  >=  .9 

6)  V041  >=  .1397 

7)  V041  >=  .7982 

8)  K041  >=  .0621 

9)  F042  -  .0701K141  >=  .8299 

10)  F042  -  .0241V031  -  .9618V041  >=  -.0859 

11)  K042  >=  .1944 

12)  K042  >=  .7496 

13)  V042  >=  .0691 

14)  V141  >=  .9 

15)  VUl  >  =  .1489 

16)  VUl  >  =  .8511 

17)  V'043  -  .1V142  >=  .8 

18)  F043  -  .078V031  -  .902V041  >=  -.08 

19)  V043  -  .994^042  >=  -.094 

20)  V043  >=  .2 

21)  V043  >=  .72 

22)  V'043  >=  .098 

23)  VU2  >=  .9 

24)  VU2-  .76VU1  >=  -.1 

25)  V142  >=  .2 

26)  V142  >=  .8 
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